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(54) Compressed video text overlay 

(57) Novel techniques for overlaying a message on-, 
to a video innage in the compressed domain;: The video] 
Image is represented as a compressed, digitize^ videa^,. 
bit stream. The message is digitized ancf comp'tessec^ . 
to form a coded message. The coded message te com-:- 



hined with the compressed, digitized video bit stream to 
^generate an; output video bit stream representing the 
... lyideq image wjth a rrnessaige overlay. The message may 
!;consis1 of tejit, alphanumeric characters, numbers, sta- 
;tus indicator^, symbols, and/or icons. 
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Description 

Background of the Invention 

1. Field of the invention , \ . ^ . s 

The invention, relates generally to' video-communi- 
cations systems, and more specifically tp^techniques tor , ' 
overlaying^ messages onto video images. 

2. Background Art 

In visual communications systems, it is possible to 
overlay a textual message onto an existing video image. 
For example, during typical network televisjon news- 15 
casts, message overlays are often used .to indicate the 
call tetters, channel assignment, and netvyork affiliation 
of a given television station, as well as the identjty and 
location of. a given newscaster. More broadly/ message, 
overlay may be performed wheriever there is a need to '20 
delivers video message vyithoqt substantially interrupt- 
ing ari ongoing yideo transmission.' ' 

One existing approach to message* overlay opsf- ' 
ates'in the analog domain. A textual. or graphical mes- ' ' 
sage, as selected by a system user, is represented usjng ' 25 
analog sigrials. These analog signals are then transrhit- * 
ted via in-band signaling or out-of-band signaling/ 
wherein the signaling band is defined with reference t6 
the frequency bandwidth of a video source signal. The ' 
overlay operation may be performed at a central control ' 30 
location or, alternatively, at a'locai terminal. For exam- 
ple, in the case of television broadcasting, text overlay ' ' 
Is performed at the television studio and/or transrnitting 
facility while; in the context of multirhedia' communica- 
tions, text overlay is usually performed at a local terminal *35 
such as a multimedia endpoint device. 

Messages transnnitted via in-band'sighalihg are di- 
rectly added to (i.e.. superimposed onto) the video' , 
source signal and, therefore, the overlay operation is 
terminal-independent. Therefore, the devices that are '40 
used to receive the messages heed not be equipped, 
with any special hardware or software. By vvay of exam- 
ple, television stations use in-band signaling somat vir- 
tually any consumer-grade television receiver will dis- . 
play the overlaid n^essage. and the viewer is not re- is 
quired to purchase enhanced equipmerit In order to vie^^ . 
the message. By contrast, rriessages trarismitted.via 
out-of-band signaling require cooperation among a plu- 
rality of message receivers. This cooperation signifies 
that the receiver rieeds to have functionality (i.e., hard- so 
ware and software) sufficient to decode the out-of-band, 
signal in order to superimpose the textual or graphical 
message with the video source signal. 

Other existing.communicatipns systems have used 
digital approaches to rriessage overlay. An input mes- ss 
sage is converted into a digital format and theri' trans- 
mitted using in-band signaling or,' alternatively, out-of- 
band signaling. For example, in the case of^ multipoint. 



multimedia video conferencing system, analog yideo 
and audio source signals are sampled and converted 
into digital form at the endpoints,,and the resulting dig-, 
itized signals are then data^conripressed for transmis- 
sion. Jhedigitizedcompre^.^ed audio and video signals 
are then sent to a multipoint \control unit (MCU).,The 
MCU. receives a digitized, compressed audio and yideo 
signal, in the form pf a compressed bitstream,.from each 
, of a plurality of endpoint devices, and the MCU provides 
proper routing for these bitstreams. To send .messages 
and to display these messages at one or more endpoint 
devices, the MCU sends corresponding control signal 
sequences, together with the actual digitized message, 
to each endpoint via a data channel. IJpon receiving the 
message from the data channel, the endpoint device 
displays the message on the local video display device. 
In thi^ example, the text overlay is endpoint-dependeht. 

Another existing technique for message^oyerlay is 
termed the digital pel domain approach. Messages are 
first cc^riverted into^ NTSC (North Ameripan Television 
Standard for (polpr).forr^^^ then' directly added to 
the analog video signal toform a composite video signal. 
The composite video signal is then digitized, corn- 
pressed, and transmitted. As the^messages are embed- 
ded into the pctual video' signal, this technique is end- 
point-independent. 

Although the foregping message overlay tech- 
niques may be adequate for some applications in broad- 
cast television anci/or multimedia conferencing, these 
techniques are not sufficient for present-day comrnuni'- 
cations systems which utilize digital signal processing 
techniques and/or digital signal transmission protocols. 
Even for communications systems that are essentially 
,analpgjn nature, such as broadcast television, digital 
processing systems are typically employed at son;ie 
point in the signal path before' the signals are converted 
back into analog form for transmission. ' 

Digital signal transmission protocols and digital sig-. 
nal processing techniques generally operate in a com- 
pressed-signaj domain, wherein an arialpg input signal^ 
is sampled, converted into digital form, and this digital , 
signal is then data-compressed prior to undergoing fur- 
therprpcessing stfps. Using conyentiprial anabg mes- 
sage, pv^rlay .techniques in conjunction., with com-, 
pressed digital signals results in inadequate system per- 
formance, Processing delays are excessive, and hard- . 
ware requirements are^urjduly burdensonrie. ] For exam- 
ple, when a compressed video source signal is inputted 
to a message:generating device, the message generat- 
ing ^evice n)'ust,Jirst cpnripletel)( decorripress the video , 
source signal intp„the p.el domain, add the desired mes- 
sage to the video source signal in the pel dorpain, and 
then comp/ess this pel-domain signal back intothe com- 
pressed dpnriain! ■ ' 

Summary of the Invention 

Novet' techniques are disclosed fcr overlaying a' ' 
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message onto a video image in the compressed do- 
main'. The video image isVepreserited as a conipressed," 
digitized video bit stream*: The messiage is digitizecJ arid 
compressed to form a' coded nHessage.' the coded mes- " * 
sage is combined with the'compress^d.'digitized video 5 
bit stream to gerierate an'o'utput vicleo bit stream repre- 
senting the video irTiage with' a message overlay. The 
message may consist of t'^kt, alphanurneric characters, 
numbers, status' indicators, isymbbis, and/or icons. 

Brief Description of the Drawings 

FIG. 'I is a hardWare blocfe diagram of a video mes- 
sage overlay system constructed according to a 
preferred embodiment disclosed herein; ' 

fig's! 2 and 3 are data structure diagrams setting 
forth illustrative coding formats for representing vid- 
eo information; 

..... ' 20 

FIG. 4 is a data structure diagram setting fpHh the .. 
'manner in which a pixel-tDy-pixei block bf an image 
is used to represent a message; and ' , 

FIG. 5 is a hardware block diagram setting forth the 
system "architecture of a multimedia "confe'rencing 
system which includes a multipoint control unit 
(MCU) having a video processor equipped to per- 
form video' message overlay according to various 
preferred embodiments disclosed herein " ' 30 

Detailed Description of the Invention 

Pursuant to various embodiments disclosed herein, 
novel message overlay techniques are described that 35 
greatly reduce processing delay and irhplernehtation 
costs. These techniques are device-independent, 1n that 
message overlays can be placed into a connpressed vid- 
eo bit stream fr-om virtually ahy videb source!' the tiech- 
niques can be advahtaQeou'sly employed /foV example, 40 
in the operational environrhent'of multipoint video com- ' 
municatibriS to pi-ovide advanced features siich ''^s * 
speaker identification, terminal name, Ipcatiohyname, 
event notification; cdriference status indicatiba'tlme of 
day, Video Signal parameters, etc. The 'Enhanced nies- ■ ^5 
sage 6v<^rlay techniques 'disclosed hereW may alsb be 
utilized in con junction 'with vide6-oh-'defYiand,' video' te- 
lephony, cable television,' satetllte" teleVlkloh,' and/or 
broadcast television. ' 

If existing techniques are used to overlay nriessages so 
onto a compressed video signal, t)ie video signal must 
be decompressed before the rhessage is overlaid, and " 
then recofnpressed after message overlay. However, 
since" the overlay operations of the embodiments dis- 
closed herein are performed in the compressed domain, S5 
the steps of decompression and recompression are 
eliminated. Such decompression and recompression " 
would add unnecessary processing delay, as well as in- 



crease the overall costs of implementation, 

Although the message overlay techniques" dis- 
closed herein are described in the operational environ- 
ment of a standard generally knoWn as the H.261 stand- 
ard, this is done for illustrative purposes only, it being ^ 
understood that these techniques are applicable *tb vir-" 
tually any video compression technique that uses a lay- 
ered syntax. Examples* of layered-syntax video' conri- 
pression tech'nicjues are the iyiPEG-1 and MPEG-2 vid-' * 
eo compression standards. The H.261 standard' is de- 
scribed in ITU-T document "Recommendation H.261, 
Video Codec for Audiovisual Services at px64'kbits/s", - 
May 1 992 and "Description of Reference Model 8", June 
9, 1989: ,^ ' ' ] 

Refer 'now to 'FIG. 1, which sets forth ',a hardware 
block diagram of a message overlay system constructed * 
in accordance with an embodiment disclosed herein. 
Message overlay system 100 consists of a hnessage 
processing unit 101, a compressed message storage 
unit 103, an overlay unit 105, a coritrol unit 107, a re- 
ceiving buffer 109, and a transrinittihg buffer 111. The 
message processing unit 1 01 includes a message to im- 
age conversion unit 113' and el message' cprhpression 
unit 115. The input 1 21 and the output 1 23 of the rries- f 
sage overlay system 100, Vjn and V_out, respectively, 
are video bit streams that are coded in coiiripliance"' with ^ 
the above-cited H.261 staridard. Input 121 receives a 
compressed video signal without rriessage overlay;' and 
output .1 23 gerierates a cbmprisssed video signal having' 
a message byeflay. ' ' . .. ( 

ln' H.26i video coding, a video iriput signal, VJn, 
represents a sequence of video frannes. Individual ^ 
frames are compressed. according to one of two types ' ^ 

of compression algorithms, ihlra-frame coding (I), and V 
predictive'coding tPj. A given video input' signal, VJn, 
may include Both I -coded, as wdjl as P-cbded, frames. ^\ 
Whether a given frame is l-coded or "P -coded 'may be - 
determined with reference to the specific coding se- • / 

quence enripibyed, as will be described in greater detail [ 
^ with i;eterence to FiGs. '2 a/id 3. FlGs. 2 and 3 are data 
structure diag'rams setting fdrtlS 'illustrative cdding for- 
mats for representing video information. 

Corhpression in the I mode includes designating 
one or rhbre video frames as I frarnes, 'dividing each of 
the I frannes into blocl^s of, for example, 8x8 pixels, and 
performing a discrete codine transform (DOT) on the pix- ' 
els in the'respective blocks.* thereafter, quantization is' 
performed on the coefficients generated by the DCt 
These quantized coefficients constitute the compressed [ 
, image information ifor the I'-encbded frame. 

Compression in the'P mode includes the steps of: ' ^ 

1) designating one or more video frames as P 
frames, and then dividing a respective P-designat- 
ed frame pf,Jor example, 8:n:8 pixels, into tour con- ' 
figuqus rriic rob locks, each 'micrpbibck having 4x4 
pixels; . , \ ^ ' • .V' /' ■' 
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2) searching the immediately preceding P or I trame ' 
for a microblock containing similar image informa- 
tion; ' " . = 

3) generating motion vectors to franslate the micrb- * ' £ 
block found in the immediately preceding I or P ^ ' 
frame to the location of the simijar microblbck in trie ' 

P frame be in c| compressed; ^ ' . ! 

4) generating a predicted frame from the immedi- io 
ately preceding frame using the motion vectors; . ' 

5) subtracting the predicted franne from'the P frame 
being compressed, on a block-by-block basis, to 
generate blocks of residues; ' " is 

6) perfornning DdT's on the blocks of residues; 

7) quaritizing the coefficients of the bidbks of DCt- 
transformed residues; and ' ' 20 

8) concatenating the quantized residue coefficients' 
and the motion vectors to form a compressed video 
sigrial.' r.- « ^ , . ^ , ....^ , 

Referring now to FIG. '2, video information V rep re- ' 
sented as a plurality of frames 201 ; 20i3, 205, 207^ 20'9, ' 
211. 21 3. 21 5, 217. 21 9,' 221 . Ea6h of these irames con- ' 
tains a repYeseritation of a two-dimensional image in the' 
form of a pixel arrky Since a given frame may represent ' 30 
a video image at a specific moment in time, a plurality ' 
of frames may be erriployed to j-epi^esent a moving^irri- 
age. Each frame represerits a picture. In an intra-frame- 
coded picture, i:e., an I picture, every microblock is intra- 
coded. That is, each microblock 'is ccxied-without refer- 35 
ence to any microblock irii the previous I or P frame. In 
a predictive-coded picture, the niicroblocks can be ei- ' 
therintra-coded or inter-coded. A plurality bf frames rep- 
resents a coded video sequence. ' ' ^ 

Each of the frames is compressed according to any 40 
one of the two types bf'compressiori "algorithms' dis- ' 
cussed above, termed l-nnode compressbn and P- " 
mode cbnripressidn. 'l-mode compression is sometimes 
referred to as intra-franrie coding (t), and P -frame com- ^ 
pression is sdmettme's referred to as predictive coding 45 
(P). In the exarriple.of FIG. 2, frames ^Oi' and 211. are 
corripressed using intra-frame coding^ and frames 203, 
205,' 207, 209.^213. 215.;217, 219, and 2i21 are com- " 
pressed using predictive coding. If'intra-f rarfie^ coding (I) ' 
is to be applied to a given frame, such as f rarrie 201 ' the so 
frame is termed an l-designated framed arid if predictive 
coding (P) is to be applied to a given fr^me. such as 
frame, 211, then the frame 'is termed a P-designated 
frame. The sequence of frames shown in FIG. 2 ©stab- ' 
lish a data structure for representing a video Jrtiage in ss 
the form of arienboded vidieb sec|uence having a plural- ' 
ity of levels arranged in a two-dim ens idna I array,' where- ' 
in each level repmsehts tfie^value of a pixel eierhent. ' ' 



This encoded video sequence may be termed a coded 
video bit stream. ^ ' . : . - 

To form the coded video bit stream for transmission, 
compressed Jmage infbrnrjation. as well as other infor- 
mation, such as motion vectors.^a^ coded using spec- 
ified code words. The code words are then rriultiplexed 
into a layered structure to form an output coded video 
bit stream. Inan H.261-like environmentVthe output cpd- 
ed video bit stream is constructed in an hierarchical for- 
mat, the data structure of which is shown, in simplified 
form, in hp. 3. , - =. • 

Considering FIG. 2 in conjunction with FIG. 3, it is 
apparent that the output coded video bit stream consists ' " 
of a serially -coded picture. Thes^uerice bf frames 201 
203, 205, 207, 209, 211, 213, 215, 21 7,* 21 9 and 221 
forms a coded video bit stream. This bit stream may be 
conceptualized as a serial representation of coded 
frames which can be processed to fornri a moving video 
.image (i.e.; a rhoving picture). A* typical sequence of 
coded' frarines is'lPPP. PIPPP wherein the letters I knd ' 
P designate l-coded and P-coded frames, respectively^ 
For ea(5h frame 221, the coded representation of the 
frame includes a header 263 as Weil as coded data 265. * 
The header 263 may include a start code and data re- ' 
lated tothe respective layer: In an 'H.i26'l systerh envi- 
ronrrieht, much of the header information is required for 
synchronization purposes. Fbr exanriple,' at the frame ' 
(picture) layer for frame 221, the picture Header '263 in- * 
eludes a picture start code (PCS) field 267, a picture' 
number (TR) field 269, a picture type field (PTYPE) 271 , 
a PErtield 273. and a PSPARE field 274. The' PEI field 
273 and the PSPA,'RE field 274 are used to store extra ' 
inserted information which can be adapted, for example. ' 
to accorfimodate future applications. ' " 

Picture data is segmented into Groups 'of Blocks 
(GOB) 223, 225, 227: 229, 231. 233, 235. 237. 239, 241 , 
243, and'245. A GOB (for ^xample/GOB 229) compris- 
es one twelfth of the coded frame (GIF) 221 picturearea ' 
Therefore/ GOB 229 nriay be conceptualized as includ- ^ 
ing one-third of one-quarter (QCIF) of a coded picture 
area. the aVed represented by one-quarter of a coded 
picture is'abbreviated as QCIF Therefore, there ab 12 ' 
GOBs 223,;225, 227, 22'9„231 , 233. 235, 237; 239, 241 , ' 
243, 245 in a CIF frame 221 . and thred'GOBs in a QCIF 
picture. The arrangerpents of GOBs iri'a GIF picture,^a^^ 
in a QCIF picture,' kfre depicted in FIG. Z ' ' 
Each GOB 229 includes header field 29 f. followed 
by a microblbck data field 298.' The'header.field 291 in- 
cludes a GOB start'code (GBSC) field' 292,' a group' 
number (GN) field 293. a group type (iGTYPE) field 294,- 
a GOB quantizer (GQUANT) field 295, and spare infbr-' 
mation fields iri" the form of a'GEI field 296 and a' 
GSPARE'field 297. Each GOB 229 consists of 33 rrii- 
crobiocks, such as "microblock 24" (reference numeraT ' 
247) arid ■"microblock 25" (reference numeral 249).'The 
arrangement of microblocks within a GOB 229 is depict- " 
ed in FIG. 2 as well as in FIG. 3. 

Each' microblbck 247 includes a header' field 275 " ■ 
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followed by a block data field 277. The header field 275 
includes a rhicroblock address (MBA) field 279. a block 
type information (MTYPE) field 281, a Quantizer typ^ . 
(MQUANt) field 283, a niotion 'vector (MVD) field 285, . 
and a coded block pattern (GBP j field 287^ The block " 5 
data field '277 of each mic rob lock 247. consists of 6 
blocks, including four luminance blocks Y1 (reference 
numeral 251), Y2 (reference numeral '252), Y3 (refer- 
ence numeral 253), Y4 '(reference numeral 254), one^ [ 
chrominance block U (reference numeral 257) and one io 
chrominance block V (reference numeral 259)" An illus- 
trative example of the contents of luminance block U 
(reference numeral 257) is set iforth in FIG. 2. Note that 
this bipck includes an 8x8 pixel array wherein all pixels ' 
have a rumjnahce value of black. ' ^ , . 

A block represents a matrix of pixels, e.g. 8x8 over . 
which a discrete cos irie transform (DCT) is perfornried. 
The array of pixels is represented . by a matrix of pixel . . 
array coefficients, including a. DC C9efficient apd a plu- 
rality of AC coefficierits. The transformed coefficient^-' 20 
(TCOEFF) 301 (FIG, 3) consists of a DCT Dp coefficient . 
(DC) occurring first, followed by respective pixel ar.ra)( 
DCT AC coefficients (AC), in the order of the>ir.relativej . . ,. 
importance. The arrangement of DC and AG coefficients . i 
in an illustrative block data field 27.7 (fjIG. 3) is shown , 
in FIG. 4. The block data field. 277 (FIG.^ 3) ponsists of / , 
the transformed coefficients (TCOEFF) 301 and an end, 
of block code (EOB) 303 which are appended at the. end, 
of each successively occurririg block of data. ^ ; ^ . 

Referring back to FIG. 1, in order to overlay a mes-, 5p 
sage onto an image in the compressed dornain. .the^.. ^ 
message (i.e., text, graphical infprnriation, icon, and/or / 
image) needs to be. converted. inta.a desired ..anci/or.. 
specified image format by the.rnessage-tonmage conT ^ 
version unit 113. Message-tOTimage conversion unit .11.3 35 
includes one or more user interface devices, and/or one 
or more datja storage devices. Messages (text, alpha- 
numeric characters, status^ indicators, icons, Jrnages, . 
and/or graphical jnforhnation) may b& stored in an elec- ^ ^ 
tronic data storage device, and selected via a us.er .ir^- ^ 40 
terface, such as a, keyboard or a corriputer rnouse, that . ' 
accesses the .data storage device. Ajternatiyely, the . . ' 
message may. be generated ,in real.time by any of the., . 
aforementioned user iriterface devices, and . optionally, 
stored in the electronic data storage device. Other ex- ^5 
amples of user interface devices include keybpards.. vid- 
eo cameras, personal computers, or the like.,, . ; 

The electronic representatiori of a nrtessage, as, re- . . 
trieved from the data storage device arid/or as gene rat- 
ed by the user interface device, may ,or , may not.be in ^ 
form that represents a videO; image. Jf this re^resent^- . 
tion does not represent an image, it nnust be converted 
into a representation of an image. However even if the.^, 
stored electronic representation, of the message does 
represent a video image, it may represent.an innage,that, . 5^ 
does not have the desired/specified fornpat. Fpf exam- . ^ . 
pie, the electronic representation of Jthe m.essage rpay , 
represent a full-sized .image that, if overlaid onto a conrv- 



pressed video signal, would cover an entire frame of that 
video signal. Or, by contrast, the electronic representa- 
tion of the video signal may represent an image that is 
quite small relative to a frame of a video signal, such 
that, if this message we^re overlaid onto the video signal, 
the rpessage would not be legible. Therefore, the mes- 
sage must.be converted into a representation of an irh- 
age, and this image must be of the proper size. relative 
to a frame of the compressed video signal oh which the 
message is to be overlaid. 

The message-to-image conversion unit 113 con- 
verts the message into the desired or specified forrnat. 
In practice, this. desired or specified format could be, for 
example, a smaller-sized version of a GIF frame of 'th^ 
yideo signal on which the message is to be overlaid, or 
a QCIF frame (GIF and QC IF have been previously de- 
fined). Therefore, the npessage-to-image conversipn 
unit 1 1 3 is adapted to convert electronic signals repre- 
senting messages (video messages, graphical informa- 
tion, text, etc.) into a digital image representation haying 
the desired/specified format of, for example, a subset of 
a GIF picture, or a subset. of a QCIF picture. 

The formatted version of the retrieved inforrnation ,^ 
is compressed into a bit stream complying with the H. i] 
261 standard by message compression unit 115. Note 
that rriessage compression unit may be ecjujpped to \^ 
compress, icons, images, symbols, and/or graphical, in- <,* 
formation in addition to textual information. The. corri- , J 
pressed tDit stream generated by message compression 
unit 115 is. stored in the.conripressed message storage , ^ 
unit 103 in the fbrnn of a coded message. .... v** 

The coded message is retrieved froni compressed. 
message storage^unit 1 03 and sent to overlay unit 1 05 4, 
when it is desire^d to. overlay a message onto.a.yideq f 
image stored in receiving buffer 1 09. Note that receiving . /.^ 

buffer 109 receives a video image from input 121 in. the ^ 
form of an input video bitstream. . . . 

The overlay unit 105, under the coritrol of ^control 
unit 107, directly superinriposes the coded . message 
stored in compressed message storage unit 103 onto a 
video bit stream jead from, the receiving buffer 109 to 
form a new, composite video bit stream which is tempo- 
rarily stored in transmission buff,er lljl. The bit stream 
stored in transmission buffer .111 is.prganized so. that. ^ 
.vyhen "the contents of the buffer are read, a composite ' 
videb bit stream is provided at output .1 23- Xhis compos- 
ite video bit stream conforms to the H.261 standard and 
has the same transfer rate as tfie input bit stream at Jnput 
121, except that, the bit stream at output 1 23 includes 
, an inserted overlay message that is not present in the 
bit stream at tl;ie input 121 , . , 

In order to ove/lay a coded message, onto an input 
video bit stream, the overlay unit 105 must first locate 
the appropriate place in the input bit stream where the 
.„coded message is to be^inserted. Once this, location is 
found, the overlay unit 105 must then peplace.a portion 
of the input, bit, stream with a coded rjiessage strearn. 
The processes of locating the appropriate place in the, . 
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bit stream, and theii replacing a portiori of the bit stream, 
may be performed on a GbB-to-G06 2'29 (FIG. 2) bakis, 
and/or on a'iriicroblock-by-microblock 247 basis.' On a" ' 
GOB-to GOB basis, the overlay unit '105' locates' the '''' 
proper'GOB 229 in the input video bit stream arid theh ''s 
replaces that GOB 229 with another GOB downloaded' 
from* the compressed text storage' unit" 103. The GOB 
229 being replaced may represent, for example, the bot- 
tom corner area of the video image stored in receiving 
buffer 109. This bottom corner area corresponds to io 
GOBS (reference numeral 237, FIG. 2) in a QCIF frame, ^ 
and GOBI 2 (reference numeral 245) in a GIF frame. To 
avoid changes in the output bit rate at outpiut 123, the 
coded messages are only 'inserted into frames that are ^ 
l-coded. For P-coded'frames, the coded messages are 75 
not inserted into these frames to form a P-coded frame 
with a message overlay. Rather, the message 1h its 1'- 
coded form is simply copied into the P frarhe'. A header 
is added to the beginning of the compressed bit stream* * 
sent to transmission buffer 111 by overlay unit 105, and 20 
the data structure of this header Specifies whether the " 
overlay was performed on a GOB-to^GOB 229 basis, or" * 
on a'microblock-by-microblock 247 basis:' ' . '^v. • 

Message conripressioh is performed by rrie'ssage ' ^ 
compression unit 115. As shown in FIG. 1 , message 2S 
compression unit 115 inciudeis a discrete cosine trans- ' 
form unit (DCT Unit) 125, a quantizer 127, and a vkridble ^ 
length encoder (VLE) 129. The DCT unit 125.' quantizer 
1 27. and VLE 1'29 are sWndardized building blocks well- - 
known to those skilled in the art. The structure and tunc- 30 
tioning of these blocks is set forth in great delall in the 
above-cited H. 261 standard. ■ • < / - • . • • ^ 

At the message conripression unit 115, the input Vid- ' 
eo message from the message to image bbnversidn unit 
1 1 3 is first partitioned into one or more 8x8 blocks. The 35 
8x8 blocks are then transformed by the DCT unit 1 25 to " 
generate corresponding blocks of 8x8 DCT coefficients. 
The corresponding blocks of 8x8 DCT coefficients are 
quantized at quantizer 1 27, and the quantized DCT co- " 
efficientsaresentto VLE 129. At VLE 1'29, thequantized 40 
DCT coefficients are coded, together'v^^ith quantized in- ' ' 
formation, using specified code words '(pursuant to H.' ' 
261 ). These coded, quantized DCT coefficients are then 
multiplexed into a layered structure such as GOBs 229 
(FIG: 2), or hnicroblocks 247, to form* a message bit 45 
stream representing a compressed, coded message. ' 
Furthermore, according to the retjuiremeht of the over- 
lay unit 105, proper header information is added to the 
message bit stream to form a complete coded represen- 
tation of the nriessages. For example, if the GOB 229 so 
level is used to define the organizational layer, a 'GOB ' 
header field 291 (FIG. 2) specifying a C30B number (i.' 
e., GOB4, reference numeral 229) is added to the mes- 
sage bit stream. In this exampie. each coded message * 
is represerited in the form' of a GOB bit stream.' These 'ss 
GOB bit streams are stored in the compressed message 
storage unit 103. = ^ i . . • : . , .^r.-. 

Note that the message pfocessihg procebure'de- " 



scribed above,' including the step^s of message-to-image 
conversion and message compression, can be per- 
formed either on-line; or off-line, depending on the spe- 
cific system applicatioh: In some applications, where the 
"desired content of the ;me.ssage is kriown Beforehand, 
then this rnessages can'' be processed off-line and 
stored in compressed message storage unit 103. Com- 
pressed m'essage storage unit 1 i33 can be'emplbyed to 
store a plurality of such messages' Individual messages 
may then be selected for overlay via an optional user 
interface device, such as a keypad, keyboard, personal' 
computer, computer mouse, and/or'touch-s'creen,' cou- 
pled to control unit 107. Based upon this user input, con- 
trol unit 107 selects the proper coded message arid 
downloads this message to thb overlay unit '105 for sub- 
sequent insertion. In this manner, processing delays are 
furthier reduced since the conversion o'f the rriessage in- 
to image tomn. and the compression of this image, are 
performed beforehand. : ' < 

Tfie' message overlay systenri' of FIG. 1 has been 
expferimentatly verifiedV^^ textual messages 

are more difficult to compress than other types of video 
signals represehiing; for exanriple, ai typical video con- 
ference^ scene:' Therefore, using a given bit rate, non- " 
textual portions* of a reconstructed image may be ac- 
ceptable? whereas the reconstructed text may appear 
somewhat blurry. A computer sinriulation was conducted 
to test the'perfbrmahce'of the system of FIG. f Where a 
textual message Is to be overlaid onto a video image. A 
sequence of digits and letters were used ^s the input 
text. The'text was converted into a subset of 'image 
blocks of 48x1 76 luminance pels and 24x88 Cr and Cb 
pels, which 'represents a GOB ai-ea in H.261 . Text was 
placed into a ihriage blockbf 32x1 60 luminance pels', and 
16x80 Cb'and Cr p'els. The simplest and most straight- 
forward way to represent each character or digit is by 
using either an 8x8 block or a 16x16 block. 

FIGV'4 shbws^the manner in which a 16x16 image ' 
block is allocated to' represent 8 digits'and 3 letters, re- 
spectively. In the case of using an 8x8 block, the t'otaL 
number of digits of characters 'that can be' represented 
is 80. and,in fhe caseof a16xi6blbck,the'totalhurriber 
of chai-acters that can be represerited is 26:' In the sim- 
ulation; a 1 6x1 6 block was ehnplbyed to represent digits ' 
and/or' letters.' ' ' ■ ' - r 

The 32>c1 60 pels image blocks were sent to riies-' 
sage compression uriir 1 1 5 (FIG . 1 ). At the compressiori 
unit 115, the 32xT60 image block Was partitioned into 
8x8 sub^ bibcks.* Each 8x8' block" w^^^^ 
transifoi-nrieci iritd OCT representations in trie DCT do- 
main by DCT unit' 1 25. The DCT coefficient's were fur- 
ther quantized at the quantizer 127. The quantizer 127 ^ 
was controlled By control 'unit 107 which, ^according to 
the input bit rate, generates appropriate quaritizaiion pa- 
rameters for th^ quantizer 127: For testin'g purposes, a ' 
variety* of different quaritizatiori pararrieters were usbd, ' 
andlhe quariti'Mtion paranrieter Itself was fixed for all 
the text blocks. The quantized DCT coefficents, together ^ 
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with quantization parameter, were sent to variable^ 
length encoder (VLE) 129. At the \/Le'i29, the quan- 
tized DCT coefficients and the quantization parameter ^ , 
are coded using code words specified in the, H.261 
standard to form an H.261 -coded micfdbipck (FIG. 2, s 
247) of data. The coded micrdbloc'k 247, and the total , 
number of bits used'to ehcode the 32x160 image block,, 
are stored in the compressed message storage unit 1 03. 
To check the'picture quality of the coded text, the recon- 
structed texts were obtained and the signa!:(o-noise ra- io 
tio of the reconstructed text was computed. ■ 

At the message .overlay unit 105, the input bit, 
stream obtained from the receiving buffer 1 09 was test- 
ed. For the first frarrie, which was an I coded frarne,.the 
bits representation of GOB 1 to GOB 11 vyere passed. 15 
by the text qverlaY unit 105 without any modification., , 
Message byerlay unit 105 modified GOB 12 (reference 
numeral 245, FIG." 2) according to the fqllowingj ryles. , 
For GOB 12, no changes are made to rrijcroblock^.l t o . 

1 2 of micrpblock field 298 (FIG. 2), and no changes are 20 
made .to micrpblocks, 1 3 and ^ 23. The microbipcH data, j ^ 
stored in compressed mes93ge storage unit,! p3 .(.F I. G , 

1 ) is obtained and used tq rep I ace mic rob locks, ,1 3 to 2^ . 
and micrqblocks 24 to 33, of micr.oblqck.field 2Q8 (FIG.. . 

2) . If the frame is a P-coded frame, motion vectors .fqr . 25 
microblocks 1 to 11, 1,2, and 23 are checked. It the^mo-.^f. ^ 
tion vectors refer to the text area, which is iri.^icroblocks . . 

1 3 to 22 and 24 to 33, the motion vectors are. all set . tq ^ 
zero. Micrqblocks 13 to 22, and 24to,33, are^then tre.at- 

ed as skipped microblocks for P-coded frarnes. proper . 30 
microblock_ addresses are produced for. the modifica-. 
tton. . . , . , : . • * i . ' ^: < 

According to an additional ennbodiment dis.clqsed 
herein, message overlay techniques are proyijded iri th^ 
operational environnnent of multipoint video, conferenq- ^ .35 
ing. Coded domain.videq message overlay can be. ap- 
plied to a wide variety of applications in visual commtji- , , 
nicati.ons. One. pf the , most important applications in- 
cludes multipoint video conferencing. In a multipoi^nt vid- ^j. 
eo c.pnferepce, for the sake of qqnvenience, partiaipants 0 
at each endpoirtt often. !wou Id like to kpow sorne. related, 
information about the.conf erencs. such. as tl;\^ r^am^ o^> . 
the current speaker,, the^ location of the current speaker 
location, the termiriat name, etc. In the enviconrnent pf; 
video conferencing, it is generally more econornical tp. ..."^^ 
overlay the necessary messages to the outgoing video 
signal at a central control point, sgch as a rriultippintcpnt.. , 
trof unit (MCU), as compared vyilh proyiding rnessage : 
overlay at an endpqint device. Moreover, providing mes- 
sage "overlay at the MCU provides enhanced flexibility 50 
as opposed to^ providing message overlay, at an end- 
point. . . ■ , . h r . M - 

FIG. 5 is a block diagrarn shovying a multipoint video > 
conferencing sy stern ^constructed ir;i accordance with an 
embodiment disclosed herein, and including an ennbed- . 55 
ded message overlay unit 560 in .MCU 6T0. /^s s.hovyn 
in FIG . 5, endpoint devices, such as f if st endpoint devicp.e 
501 . second encjpoint device 502, and third endpoint de,- , 



vice 503, are. coupled to MCU 510 via ISDN network 
504. Th.ese endpoint devices include one or more user 
interface devices as, for example,, a video display, a key- 
board, a microphone, . a spealser, and/or a videp.carnera. 
.ISDN network 504 is shown fqrjllustratiye purposes, it 
being understood that a network other than an ISDN net- 
work, and/qr; anqther type of network in. addition to an 
ISDN network, may be employed.. 

The endpoint devices (501 , 502, 503) are adapted 
,to communicate, using an existing rriultimedia commu- 
nication protocol. (or protocols), such as ISDN. The end-^ 
point, device multimedia comnriunicatiqn protocol con- 
trols the presentation of media streams (electronic sig- 
nals representing ^udio., video, and/or data information), 
to the endpoint device user Endpoint devices may.f unc- 
tion bi-directionally,, both, sending and receiying multi- 
media information, or, , alternatively, endpoint devices 
may function unidirectionally, receiving, but not sending, 
multimedia , information,, or sending,, but not receiying , 
. multimedia information. An example of a suitable^ end- 
poirnt device i^ an ITU7.T H.3^Q audiovisual terrriinal. but, 
in general, a device capable of, terminating a digital mul- 
timedia stream and,presenting. it to the issej is considr 
ered an.endpoint,deyice.. A particular pro?:Juct e^xample 
.of aQ H.320 endpoint (;levice is the AT&T GjS Vistium. 
device. off e^red for sale, by Lucent Technolqgies of j^urp, 
ray Hill,.New Jefsey. . . t . . j; .. : > l -n-.. 

MCU 510 is . a conn puter-cont rolled device which in- 
cludes a multipjicity of corpmunicat ion ports, such-, as 
first communicatipnsjport 570 and secpr;i,cj cpmrnunica^. 
tions port 572,; These communications ports are adapt- 
ed for selective interconnection, .in a. variety of ways to 
provide communications .e^mong a group of endpoint .de- 
vices. Eacin. of a'PluraHty, of endpoint devices are,cpur 
pled to the MCU- through a cprrespondipg commijnica- 
tion port. MCIJ '501 also incjudes a control unit ip7,^an 
audio processing unit 541 , a video processing unit 542, 
and a data .processing, unit 543. The^contrpl unit 540, 
audio processing ur}it 541, yideo processing unit 542,, 
and d^ta processing unit ;543, each contain a nnicroproc- 
essor device. Howeyerjtiis also possible fortvyq or more 
of the, aforementioned processing. units to share a single 
microprocessor deyiqe. , .* ' ; 

The control un.it 107 is coupled to a message, 
processing unit 101, and. the message. processing. unit 
101 is coupled tq a compressed message storage, unit 
103. The rriessage processing unit 10;! is controlled by 
a microprocessor device. which. may be integral tp the 
message processing unit, and/or within rnessage over- 
lay unit 1,05, and/or within . video processing, unit 542, 
and/or, within compressed nriessage storage unit 103. 
The compressed message- storage unit 103 includes a 
mennory. device, such as any desired conibination, of 
random-access memqry (RAM), read-only memory 
, (ROM), one or more data-storage drives, or the like. Vid- 
eo pi;ocessing unit.542 includes ^.message overlay -unit 
105. Note that the structure and functionality of. mes- 
sage processing unit 101 v compressed message stqr- 
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age unit 1 03, message overlay unit 1 05. and control unit 
107 have been described above in connection with FIG 

Each communications port 570, 572 includes a net- ' ' ' 
work ihterfaci35il, a demultiplGxer513,^nda multiplex-' ' s 
er 522. A common' internal switch 5i'9 is used to selec-'"' ' 
tively interconnect (a) demultiplexers of communica- • 
tions ports' (including, for example, demultiplexer 513),' ^' 
(b) multiplexers of communications ports (including', for ' ' 
exanriple, multiplexer 522), (c) cohtrbl unit 1 07, (d) audio fo 
processing unit 541, (e) video processing unit 542, in- 
cluding ' message' overlay unit 105, and/or (f) data " 
processing unit 543; Network interface 511 includes cir- 
cuitry for implementing a conversion function between 
line signal coding, which is a well-known coding tech- 15 
nique, and Px64 kbps H. 221 -encoded signals, which 
are well-known and described in the widely-available H. 
221 standard. The output of network interface 511 is an 
H.221 -encoded signal which actually represents a mul- 
tiplexing of several different types of information (audio, 20 
video, data, and control); therefore, network interface 
511 must send the incoming MCU H.221 stream to a 
demultiplexer 513. Likewise, network interface 511 re- 
ceives the outgoing MCU H.221 bits stream from a mul- 
tiplexer 522, which combines the individual processed 25 
signals for a particular endpoint device '501. ^ • 

The demultiptexer'51 3 separates an incoming mul- 
timedia signal stream into four components"; a first com- 
ponent including electronic signals representing control, 
a second component including electronic signals' repre- 30 
senting audio, a third component including electronic 
signals representing video, and a fourth component in- 
cluding electronic' signals representing -data. The four 
outputs of demultiplexer 51 3 are fed to common internal 
switch 519. . . . : , 35 

Multiplexer 522 accepts a jDlurality of incoming mul- 
timedia signal component from the ^common internal 
switch, such as a first component representing control, 
a second component representing' audio, a third com- 
ponent representing video, and a fourth corhponent rep- 4o 
resenting data. The rnuttiple)<er 522 integrates these 
four components onto a single multimedia signal stream 
and sends the multimedia signal stream to the network 
interface 511, which in turn, routes the signal stream to 
the proper endpoint device 502. Note that the f6ur output 45 
signals from a demultiplexer 51 3 of a given communi- 
cations port 570; representing control, audio, video, and 
data, may be inputted to respective inputs of a multiplex- 
er 522 of another communications port 572 by common 
internal switch 51 9. so 

Common internal switch 519 contains a plurality of 
electronic switches, buffers, and amplifiers under the 
controlof control uhit 107. Cbnnimon internal switch 519 
is coupled to an audio processing unit 541 for mixing 
and switching electronic signals representing audio, ss 
Corhrhon internat switch 51 9 is "also coupled to a video 
processing unit 542, to a control unit-107, arid to a data 
processing unit 544 for mixing and switching electronic 



signals representing video',' control, and data; res'p'ec- 
tively: The common* internal switch 519 receives four 
outputs 51 4, '51 5, '516 517 from a giv^en communications 
port 570. and routes each of 'these four outputs to the 
corresponding four inputs 528', 529. 530, 531 of another 
communications port S72, siifch th'at a control signal out- 
put 514 is connected'to a control Signal input 528, ah 
audio signal output 51 5 is connected to an audio signal 
input 529, a video signal output 516 is connected to a 
video signal input 530, and a data' signal output 517 is 
connected to a data signal input 551. The common in- 
ternal switch 519 also routes four inputs 518, 519, 520 
521 of a given commuhicatiohs port 570 to the corre'-' 
sponding four outputs 524, 525, 526, 527 of another 
communications port 572, such that a control signal out- 
put 524" 'is cdnriected to a control Signal input 518, an- 
audio signal output 525 is connected to ah audio signal 
input 519, a video signal output 526 is connected to a' 
video sigrial input 520, and a data signal output 527 is 
connected to a data' signal input 521 • 

Audio processing unit 541 is a^unit of software and 
hardware for processing audio signals. The processing 
may take the forrti of switching the aUdio, mixing the au- 
dio, or both. Ih thecase of audio mixing, the input signal 
to the processing unit is the aggregated aiidio'signal, : 
which consists of audio signals from all of the commu- 
nications ports 570, 572: For an N -port MCU 5T0; this 
signal includes N audio signals from the demultiplexers 
51 3 of each of N communications ports 570, 572. To mix 
the audio, audio processing unit 541 decodes the audio 
input received from common internal switch 51 9. adds 
the linear signals obtained by decoding, and then re- 
encodes the 'linear sum appropriately for each endpoint 
device* 501 , 502,' 503. The summation may be limited in ' 
any of various ways: the output sum for a given endpoint 
device 501 may exclude that endpoinfs audio input; the ' 
sum may include inputs whose present or recent past 
aLidioV amplitude values ' exceed a certain amplitude 
threshold; or the suni may be'controlled by dneperson 
using some method of chair-conti-ol. The obtput of the 
audio^rocessihg unit 541 , in this case, is N processed 
audio signals. In the case where audio processing unit 
541 is implemented using a siinple audio switch, the in- 
put sighahs a single audio signal selected based on the 
control signals from^the control unit 107. There is no au- ' 
dio probessing in's'uch a' case. The audio input is broad- 
cast tdall other audio processing unit'54l outputs; either ■ 
automatically or Ondermahuar cohtrbl. ' - ' 

Data processing unit 543 contains one or both of 
the functions generally known to those skilled in the art 
as "broadcast" or "MLP". For each' type of broadcast da- ' 
ta, data input is accepted from only one endpoint device' 
501 'at any one time. Therefore; the input signal is the 
data output fr-bnn one of the communications ports* 570. 
The accepted data streams are broadcast to other' end- 
point devices '502, as determined by the control unit 1 07 
according to the Capability of the connected endpoint 
devices 502; 503 to receive such data. ' ' 
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Control unit 107 is responsible. Ipr. determining the 
correct signal routing, mixipg, switching, format and tim- . 
ing of the audio, video, data and control signals through-. . 
out a . conference. The conlrol unit 107 also controls 
mode switching on outgoing streanp.5 to accommodate, 
combinations of audio, vjdeo, and d^ata. according to the . 
declared capabilities of the connected endpoint devices 
501,. 502, 503. . . '->.,..'- ■ ... . 

The inputs to the control unit 107 are chair-control 
commands, anc! commands embedded in H. 221 bit 
streams. Cqrijimands from endpoint devices 501,, 502, . 
503 are routed to the control unit 107 to ensure the cor- 
rect distribution of the bit streanns to. the audio, video, 
and data, processing units 541, 542, 543, respectively. 
Control .unit 107 also ensures that the correct audio de- 
coding algorithm i?..used at the inpiits to the. audio 
processing unit 541 , and that any incoming^data is sent, 
to data processing unit 543, and/or .to an MLP, processor 
within control unit 107, Tfp^ controlunit 107. also directs, 
the switching of bit streams from the audio, video, and. 
data processing units 541 , 542, -543, respectively,- to 
each nriultiplexer 522, and specifier the audip encodlag 
algorithm to. be used in an audio mixer within the audio., 
processing unit 542, and at each outp,ut.tromitt;iis audio 
mixer. The bit streams are, routed to and f rorn.the various, i 
processing units 541 , 542, 543 by the common- internal 
switch 519, and this, common internal switch 51 9 .is m- 
der the .control of control unit 107.: . , . : , 

f\/lessage processing unit. 101 , cpMpled to control ; 
unit 107, accepts messages, .which may. be? in Jextua.l. 
format, and converts these me.ssages into encoded, bjt^, 
streams. The.encoded bitstreams are stored in thede^n 
ignated locations in the compressed message ^storage, 
unit , 103. Under the control of. the control unit 107; se-,. 
lected coded, compressed messages are downloaded 
from message storage, unit 103 to a .message overlay 
unit .105 within video processing vnit 542. . 

Video processing unit 54?, which includes roes,sage ; 
overlay.unit ■)05. processes video signal Is received frorp:. 
the common internal svyjfch 519. The processing .may 
take the form of switching the video, matching the video, u 
or bot|7. - , . . ■ i.. : - 

In vjdeqs.vyitching, the video processing unit 5,4? .rer , 
ceives a selected video signal from the comiTJpn internal 
switch 51 9, overlays thepqded nciessage. obtained .from 
the compressed ^ message storage unit .103-. .^r\0 trans-, 
mits the overlaid video; signal to some or all qtherven.d-. 
point devices 502. In video matching, fhQ yidep.pirocess- , 
ing unit 542 receives, more than one.^electeci video p\g- 
nal from the common internal switch^ 5.1 9. The§apriultiple, 
video signals are composited tp form a single composite 
video bit stream.. A rnessage. frqm cpmpresseid, nr\es- 
sage storage un.itj103,can beoyerlaid onto the individual 
video signals, or.lhis m^^sage can- be .overlaid onto the, 
composite yideo bit .stream., ... ... 

Selection pi, the video sjgnal that cprmnnonjnternal 
switch 51 9 will send to a giv.en encippint devjqe 503 nnay 
be automatic and/or under manual control.; For instance,; 
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the audio processing unit 541 and the video processing 
unit 542 may be autornatically controlled,, such that the 
present speaker receives the picture of the previous 
speaker while all other terminals receive the picture of 
the present speaker. A time delay .may be. incorporated, 
into the- switching to avoid, excessively frequent image 
changes caused by spurious sounds. As with the audio,, 
video switching may also.be controlled directly by a per- 
son through chair-control. If the processing delay pro- 
vided the video processing unit 542 and the processing 
delay of the audio processing unit 541 differ by a. signif- 
icant amount, a ^compensating delay may be inserted 
into either the audio signal, or the video signal, to retain 
lip synchronization., , . . . 



Claims 

1 . A method for overlaying, a message onto a video 
image in the compressed domain, comprising the 
following steps; . . . 

. . i (a) representing . 3 video image , as . a com- 
i . , pressed, digitized video bit stream; 

(b) generating a representation.pf a video mes-, 
: sage in digitized, form to provide.a digitized vid- 
eq message; r_ . . 

v(c) compressing the digitizedytdep message to 
form a. coded yideOfme^sage;iand .^^ . 

• (d). combining ftha poded video message with 
the compressed, digitized video bit, stream to 
generate an output video bit stream represent- 
ing the video irnage with a. message overlay in,- 
cluding the video message. , 

2. The method qf claim, 1 vyherein the.yideo message 
incJudeS) representations of text, ; alphanunneric 
characters; numbers, status indicators, , symbols, 
and/q.r icons. ^ , , ^ . . . . 

3. Apparatus. for overlaying a video naessage onto a 
video irriage in the compressed domain connprising; 

(a) a videa processing device, for representing 
. a video image as a compressed, digitized yideo 

bit stream; j,, ^ . . ; . 

(b) a message generation device for generating 
a representation of.a video, message in digitized 
form to, provide a digitized video, message; 

.(c) a - video compression device for cqrn press - 
ing the digitized video m.essage to fo.rrri a coded 
. .. video message; arid ... -i 
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(d) a mixing device for combining the coded vid- 
eo message with the compressed, digitized vid- 
eo bit stream to generate an output video bit 
stream representing the video image with a 
message overlay including the video message. s 

The apparatus of claim 3 wherein the video mes- 
sage includes representations of text, alphanumer- 
ic characters, numbers, status indicators, symbols, 
and/or icons. ' fQ 
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